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SUMMARY 

Problem 

The  Computer  Managed  Instruction  system  developed  at  Memphis  provided 
feedback  on  tests  by  means  of  delayed  computer  printouts.  There  was 
a  variety  of  problems  associated  with  this  method.  The  purpose  of  this 
study  was  to  explore  the  use  of  chemically  treated  answer  sheets  as  a 
means  for  solving  some  of  these  problems. 

Background  and  Requirements 

The  chemically  treated  answer  sheets  allow  the  student  to  see  whether 
his  answer  is  correct  immediately  after  he  makes  a  response.  Further, 
the  answer  sheet  is  self  scoring.  Thus,  it  can  be  used  to  provide  an 
immediate  indication  of  the  areas  in  which  the  student  needs  remedia¬ 
tion.  It  seemed  pertinent  to  test  the  value  of  these  features  in  a 
training  situation.  A  portion  of  the  AMFU  (Aviation  Mechanical 
Fundamentals)  Course  was  used  for  this  purpose. 

Approach 

The  performance  of  a  group  of  students  who  used  the  chemically  treated 
answer  sheets  in  part  of  the  AMFU  course  was  compared  with  the  perform¬ 
ance  of  a  group  who  used  optically  scannable  sheets  in  learning  the 
same  material.  The  two  groups  were  compared  in  terms  of  (1)  the  time 
required  to  complete  the  portion  of  the  course  where  the  chemically 
treated  sheets  were  used  and  (2)  the  number  of  errors  made  on  those 
items  from  the  end-of-course  examination  which  pertained  to  material 
covered  during  that  portion  of  the  course. 

Findings 

The  group  using  the  chemically  treated  answer  sheets  was  found  to 
require  reliably  less  training  time  in  the  portion  of  the  course  which 
was  evaluated.  Training  time  was  reduced  about  15%  by  means  of  the 
chemically  treated  answer  sheets  and  associated  changes  in  procedures. 
There  was  no  statistically  significant  difference  in  the  number  of 
errors  made  by  the  two  groups  on  the  portion  of  the  final  examination 
used  as  a  criterion. 

Conclusions  and  Recommendations 

The  results  of  the  present  study  indicate  that  use  of  chemically 
treated  answer  sheets  leads  to  a  decrease  in  time  taken  by  students  to 
reach  a  predetermined  criterion  level.  In  view  of  the  reduction  in 
training  time,  the  ease  in  test  scoring,  and  the  wide  range  of  applica¬ 
tion  for  the  paper,  it  is  recommended  that  existing  instructional 
programs  be  closely  examined  with  regard  to  the  possible  advantages  of 
incorporating  chemically  treated  answer  sheets  into  the  programs. 
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AN  EVALUATION  OF  THE  USE  OF  CHEMICALLY  TREATED  ANSWER  SHEETS 

A.  Problem 

The  present  study  was  undertaken  in  an  effort  to  assess  the 
feasibility  of  using  chemically  treated  answer  sheets  within 
courses  developed  as  part  of  the  CMI  (Computer  Managed  Instruc¬ 
tion)  Project  at  the  Naval  Technical  Training  Command,  Memphis. 

The  chemically  treated  answer  sheets  are  designed  so  that 
marks  with  a  special  marker  will  uncover  the  latent  image 
answers  printed  on  the  sheets.  Thus,  the  student  sees,  im¬ 
mediately  after  making  a  response,  whether  his  answer  is  correct. 
If  the  response  is  incorrect,  he  may  then  mark  other  alterna¬ 
tives  until  he  finds  the  correct  answer.  Thus,  the  student  is 
provided  with  immediate  knowledge  of  the  correct  answer  to  each 
question,  but  there  is  no  way  for  him  to  look  ahead  or  cheat 
before  recording  his  answer.  Since  the  answer  sheet  is  self 
scoring,  it  can  be  used  to  provide  an  immediate  indication  of 
the  areas  in  which  the  student  needs  remediation.  A  sample 
sheet  (reduced  in  size)  .is  reproduced  in  Appendix  A. 

The  value  of  these  features  was  determined  by  comparing  the 
performance  of  a  group  of  students  who  used  the  chemically 
treated  answer  sheets  in  part  of  the  AMFU  (Aviation  Mechanical 
Fundamentals)  Course  with  the  performance  of  a  group  of  students 
who  used  optically  scannable  answer  sheets  in  learning  the  same 
material . 


B.  Approach 

1 .  General  Procedures 

The  course  is  broken  down  into  blocks  of  instruction  called 
modules.  When  a  student  completes  a  module,  he  takes  an  ex¬ 
haustive  test  that  covers  every  objective  in  the  module.  If 
he  meets  criterion  on  each  of  the  general  areas  (or  groups  of 
objectives)  covered  by  the  test,  he  proceeds  to  another  module. 
If  he  fails  to  meet  criterion  on  one  of  the  general  areas,  he 
is  assigned  a  remedial  test  that  covers  that  area.  He  does 
whatever  remedial  work  he  feels  to  be  required  in  order  to  cor¬ 
rect  his  deficiency,  then  takes  the  remedial  test.  When  he 
has  passed  tests  on  all  modules  within  the  course,  he  takes  the 
end-of-course  examination. 

2 .  Control  Group 

Students  in  the  Control  Group  were  assigned  several  modules 
at  a  time.  It  was  suggested  that  the  students  work  through  the 
modules  in  the  order  in  which  they  were  listed,  but  the  stu¬ 
dents  were  actually  free  to  work  on  any  of  the  assigned  modules. 
The  students  in  this  group  responded  to  their  tests  on  optical¬ 
ly  scannable  answer  sheets.  When  they  finished  a  test,  they 
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deposited  the  answer  sheet  in  a  basket  for  subsequent  pickup  by 
the  computer  operator.  The  student,  meanwhile,  continued  with 
other  assignments.  The  answer  sheets  were  collected  every  20 
minutes  for  evaluation  by  the  computer.  The  average  delay  be¬ 
tween  the  completion  of  a  test  and  the  receipt  of  feedback  from 
the  computer  was  about  30  to  40  minutes. 

If  a  student  did  not  meet  criterion  on  the  initial  test  fol¬ 
lowing  a  module,  the  feedback  consisted  of  a  list  of  the  general 
areas  on  which  he  had  not  met  criterion  and  a  remedial  test  as¬ 
signment  for  each  of  these  areas.  If  he  did  meet  criterion,  he 
received  no  feedback.  Feedback  on  the  remedial  tests  consisted 
of  (1)  a  list  of  any  items  he  might  have  missed  and  (2)  either 
a  statement  that  he  had  met  criterion  or  another  remedial  test 
assignment . 

3 .  Experimental  Group 

Students  in  the  Experimental  Group  used  the  chemically 
treated  answer  sheets,  but  only  during  the  first  half  of  the 
course;  during  the  second  half  they  used  the  optically  scan- 
nable  answer  sheets.  When  the  student  finished  one  of  the 
chemically  treated  answer  sheets,  he  took  it  directly  to  a 
monitor  for  evaluation.  If  he  did  not  meet  criterion  on  one  of 
the  general  areas,  he  was  told  the  area  in  which  he  was  de¬ 
ficient  and  was  assigned  a  remedial  test  on  that  area;  if  he 
did  meet  criterion  on  all  areas  within  a  module,  he  was  as¬ 
signed  a  new  module.  Since  there  was  little  or  no  delay  be¬ 
tween  the  completion  of  a  test  and  feedback  on  the  test,  stu¬ 
dents  were  assigned  only  one  module  at  a  time. 

4 .  Subjects 

The  Experimental  Group  consisted  of  19  Navy  enlisted  men 
who  went  through  the  CMI  version  of  the  AMFU  course.  The 
Control  Group  consisted  of  46  Navy  enlisted  men. 

5 .  Method  of  Evaluation 

The  two  groups  were  compared  in  terms  of  (1)  the  time  re¬ 
quired  to  complete  the  first  half  of  the  course  and  (2)  the 
number  of  errors  made  on  those  items  from  the  end-of-course 
examination  which  pertained  to  material  covered  during  the  first 
half  of  the  course.  The  latter  consisted  of  233  of  the  490 
end-of-course  test  items.  These  items  provided  an  extensive 
coverage  of  all  objectives  taught  during  the  first  portion  of 
the  course.  The  difference  on  each  criterion  was  tested  by 
means  of  an  analysis  of  covariance  in  which  a  composite  of 
scores  from  the  Navy  Basic  Test  Battery  was  used  as  the  con¬ 
trol  variable.  The  composite  was  determined  on  the  basis  of 
data  collected  from  previous  students  in  the  course. 
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C.  Results  and  Discussion 


Table  1  contains  both  the  mean  and  adjusted  mean  training 
time  (in  hours)  for  each  group  in  the  first  segment  of  the 
course.  Table  2  contains  the  analysis  of  variance  and  co- 
variance  on  these  same  data. 


TABLE  1 

Mean  and  Adjusted  Mean  Training  Time  for 
Groups  E  and  C  in  Course  Segment  I 


Original  Means 

Adjusted  Means 

E  21.8 

21.5 

C  25.  0 

25.3 

Difference  -3.2 

-3.8 

TABLE  2 

Analysis  of  Variance  and  Covariance  on  Measures  of 
Training  Time  for  Segment  I 


Source  of 
Variation 

SS 

df 

MS 

F 

Groups 

Error 

Total 

138.3303 

1526.3932 

1664.7235 

/ 

1 

63 

64 

138.3303 

24.2285 

26.0113 

5.7094* 

Groups  (adj.) 

191.9713 

1 

191.9713 

9.0776** 

Error  (adj . ) 

1311.1624 

62 

21.1478 

Total  (adj . ) 

1503.1337 

63 

23.8593 

Notes  — 


*  Significant  at  .05  level  of  confidence 
**  Significant  at  .01  level  of  confidence 
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The  Experimental  Group  was  found  to  require  reliably  less 
training  time  in  the  first  segment  of  the  course  than  the 
Control  Group,  both  before  and  after  adjustment  for  differences 
in  aptitude.  The  data  indicate  that  training  time  was  reduced 
about  15%  by  means  of  the  chemically  treated  paper  and  as¬ 
sociated  changes  in  procedures. 

Table  3  contains  both  the  mean  and  adjusted  mean  number  of 
errors  on  the  final  examination  for  each  group  on  the  material 
taught  in  the  first  segment  of  the  course.  Table  4  contains 
the  analysis  of  variance  and  covariance  on  these  data.  There 
was  no  statistically  significant  effect,  either  before  or  after 
adjustment. 


TABLE  3 

Mean  and  Adjusted  Mean  Number  of  Errors  on  End-of-Course 
Test  for  Groups  E  and  C  on  Material  Taught  in  Course  Segment  I 


Original  Means 

Adjusted  Means 

E 

13.8 

13.4 

C 

11.3 

11.7 

Difference 

2.5 

1.7 

TABLE  4 

Analysis  of  Variance  and  Covariance  on  End-of-Course 
Error  Scores  for  Segment  I 


Source  of 


Variation 

SS 

df 

MS 

F 

Groups 

81.5949 

1 

81.5949 

1.5830 

Error 

3247.2666 

63 

51.5439 

Total 

3328.8615 

64 

52.0135 

Groups  (adj.) 

35.0635 

1 

35.0635 

.7611 

Error  ( adj . ) 

2856.2893 

62 

46.0692 

Total  ( adj . ) 

2891.3528 

63 

45.8945 

D.  Conclusions  and  Recommendations 


The  results  of  the  present  study  indicate  that  use  of 
chemically  treated  answer  sheets  led  to  a  decrease  in  time  taken 
by  students  to  reach  a  predetermined  criterion  level.  No  effect 
on  a  criterion  referenced  end-of-course  test  was  detected. 

It  is  possible  that  the  reduction  in  training  time  could  be 
due  as  much  to  the  sequential  administration  of  materials  as  it 
is  to  the  chemically  treated  answer  sheet  itself.  However,  the 
sequential  administration  (where  the  student  was  forced  to  meet 
criteria  on  one  subject  area  before  continuing  to  another)  was 
made  administratively  possible  by  the  self  scoring  property  of 
the  chemically  treated  answer  sheets.  Also,  the  students  were 
not  allowed  to  proceed  until  they  had  found  the  correct  answer 
to  each  question.  This  particular  feature  serves  as  a  signifi¬ 
cant  aid  in  the  debugging  of  tests. 

The  immediate  feedback  provided  to  the  student  is  another 
advantage  of  using  the  chemically  treated  paper.  Without  such 
item  by  item  feedback,  the  student  has  to  return  to  the  test  after 
an  elapsed  period  of  time,  reconstruct  the  question  and  how  he 
derived  a  particular  answer,  and  then  locate  the  correct  re¬ 
sponses  within  the  appropriate  instructional  material.  In  this 
respect  the  chemically  treated  answer  sheets  aid  the  instructional 
process  and  contribute  to  the  savings  in  time. 

A  larger  number  of  personnel  was  required  for  running  the 
Experimental  Group  than  for  the  Control  Group.  The  additional 
personnel  were  needed  for  checking  test  performance  against 
established  criteria,  recording  test  results  and  assignments, 
and  making  new  or  remedial  assignments.  However,  the  chemical¬ 
ly  treated  answer  sheets  used  in  conjunction  with  computer  as¬ 
signment  and  evaluation  eliminates  the  need  for  such  extensive 
recordkeeping  and  manual  assignment. 

Notes  on  the  cost  of  using  chemically  treated  answer  sheets 
may  be  found  in  Appendix  B. 

In  addition  to  the  reduction  in  training  time  associated 
with  the  use  of  the  chemically  treated  answer  sheets  and  the 
ease  in  test  scoring,  the  paper  seems  to  have  a  wide  range  of 
application  and,  in  general,  is  favorably  accepted  by  students 
and  instructors  alike.  Thus,  it  is  recommended  that  a  close 
look  be  taken  at  existing  instructional  programs  with  regard  to 
the  possible  advantages  of  incorporating  chemically  treated 
answer  sheets  into  the  instructional  and/or  testing  phases  of 
the  programs . 
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APPENDIX  A 


Sample  CMI  Answer  Sheet 


NAVY  CMI  ANSWER  SHEET  CMM4 


NAME  - 

LAST 

MIST 

INlTlAi 

•AMK/IATE 

CO/IAT  • 

JULIAN  DATE 

SCOIE 

TUI  fCMM 

SC  MOOt 

CLASS  MO 

mSTiucrot 

THE  LETTER  CZH  WILL  APPEAR  WHEN  YOU  MARK  THE  CORRECT  ANSWER 


W 

1 

2 

3 

4 

5 

w 

1 

2 

3 

4 

5 

W 

t 

2 

3 

-  4 

'  0 

□ 

□ 

□ 

□ 

□ 

18  D 

□ 

□ 

□ 

□ 

□ 

35  0 

□ 

□ 

□ 

□ 

w 

1 

2 

3 

4 

5 

W 

1 

2 

3 

4 

5 

ft 

1 

2 

3 

4 

2  0 

□ 

□ 

□ 

□ 

□ 

19  D 

□ 

□ 

□ 

□ 

□ 

36  0 

□ 

□ 

□ 

□ 

w 

1 

2 

3 

4 

5 

ft 

1 

2 

3 

4 

5 

w 

1 

2 

3 

4 

3  0 

□ 

n 

□ 

□ 

□ 

20  0 

□ 

□ 

□ 

□ 

n 

37  D 

□ 

□ 

□ 

□ 

w 

1 

2 

3 

4 

5 

W 

1 

2 

3 

4 

& 

ft 

1 

2 

3 

4 

4  3 

□ 

□ 

□ 

□ 

□ 

21  0 

□ 

□ 

□ 

□ 

□ 

38  l 

□ 

□ 

□ 

□ 

w 

1 

2 

3 

4 

5 

V 

1 

2 

3 

4 

s 

ft 

1 

2 

3 

4 

5  0 

□ 

□ 

n 

□ 

□ 

22! 

□ 

□ 

□ 

□ 

□ 

39 

□ 

□ 

□ 

□ 

w 

1 

2 

3 

4 

5 

w 

1 

2 

3 

4 

5 

ft 

1 

2 

3 

4 

6  P 

□ 

□ 

□ 

□ 

□ 

23  D 

□ 

□ 

□ 

□ 

□ 

40  0 

□ 

□ 

□ 

□ 

w 

1 

2 

3 

4 

5 

w 

1 

2 

3 

4 

5 

w 

t 

2 

3 

4 

7  I 

□ 

□ 

□ 

□ 

□ 

24  0 

□ 

□ 

□ 

□ 

□ 

0 

□ 

□ 

□ 

□ 

W 

1 

2 

3 

4 

S 

W 

1 

2 

3 

4 

S 

w 

1 

2 

3 

4 

»  0 

□ 

□ 

□ 

□ 

□ 

25  Q 

□ 

□ 

□ 

□ 

□ 

K  0 

□ 

□ 

□ 

□ 

W 

1 

2 

3 

4 

5 

w 

1 

2 

3 

4 

5 

ft 

1 

2 

3 

4 

9  il 

□ 

□ 

□ 

□ 

□ 

26  0 

□ 

□ 

□ 

□ 

□ 

43  l 

□ 

□ 

□ 

□ 

W 

1 

2 

3 

4 

5 

W 

1 

2 

3 

4 

5 

ft 

t 

2 

3 

4 

io  0 

□ 

□ 

□ 

□ 

□ 

27  D 

□ 

□ 

□ 

□ 

□ 

44  0 

□ 

□ 

□ 

□ 

w 

1 

2 

3 

4 

5 

-w 

1 

2 

3 

4 

S 

w 

t 

2 

3 

4 

it  0 

□ 

□ 

□ 

□ 

□ 

28  0 

□ 

□ 

□ 

□ 

□ 

45  0 
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APPENDIX  B 


NOTES  ON  THE  COST  OF  CHEMICALLY  TREATED  ANSWER  SHEETS 


Meaningful  cost  estimates,  of  course,  are  dependent  upon 
the  amount  of  chemically  treated  paper  required  for  a  course 
and  the  volume  of  paper  to  be  ordered. 

Students  going  through  the  course  under  consideration  in 
the  present  study  use  an  average  of  64  sheets  for  the  complete 
course.  This  includes  remedial  tests  taken  after  the  end-of- 
course  test.  The  average  time  taken  to  complete  the  course 
(and  end-of-course  remedials)  is  6  days. 

Costs  for  reorder  of  the  chemically  treated  optical  scan 
sheets  are  as  follows:  300,000  sheets  may  be  ordered  for  a 
cost  of  $24.50  per  thousand  sheets  (2.5  cents  per  sheet); 

1,500,000  sheets  may  be  ordered  for  $16.04  per  thousand  (1.6 
cents  per  sheet).  These  figures,  of  course,  do  not  include 
the  plate  costs  or  costs  for  preparation  of  camera-ready  copy. 
However,  these  latter  are  one-time  costs  which  would  not  be  in¬ 
volved  in  a  reorder. 

The  costs  for  an  initial  order  of  chemically  treated  answer 
sheets  can  be  broken  down  as  follows: 

1.  Preparation  of  camera-ready  copy  is  $175  for  each  basic 
form  (i.e.,  for  each  answer  sheet  design)  and  $23  for  each  in¬ 
dividual  answer  pattern.  In  the  present  study  16  different 
answer  patterns  were  prepared.  For  each  pattern,  five  latent 
letters  were  randomly  distributed  among  the  alternatives  for 
each  item  on  the  basic  form.  Each  of  the  five  letters  could 
then  be  used  as  a  key  for  correct  answers.  Therefore,  each 
individual  answer  pattern  provided  five  different  correct 
answer  patterns.  The  basic  design  also  had  columns  which  could 
be  optically  scanned. 

2.  Plate  costs  are  approximately  $120  for  two  plates,  one  plate 
for  the  visible  ink  and  one  for  the  latent.  Four  8  1/2"  X  11" 
sheets  are  printed  per  impression.  Thus,  four  individual  answer 
patterns  may  be  obtained  from  two  plates. 

3.  The  cost  of  1,000  crayon  imaging  markers  is  approximately 

$100. 

The  following  cost  estimates  are  for  optical  scan  sheets  which 
do  not  provide  immediate  knowledge  of  results  (i.e.,  are  not 
chemically  treated) : 

1.  Sheets  printed  on  one  side  only  are  $100  for  the  first 
thousand  copies  and  $6.05  for  each  additional  thousand.  (If 


9 


camera-ready  copy  is  provided,  the  $100  estimate  for  the  first 
thousand  would  be  reduced  to  $35.)  The  cost  per  sheet  on  large 
runs,  therefore,  would  be  $6.05  y  1,000  or  .6  cents  per  sheet. 

2.  Sheets  printed  on  both  sides  are  $165  for  the  first  thousand 
copies  and  $7.59  for  each  subsequent  thousand.  (If  camera-ready 
copy  is  provided,  the  $165  estimate  would  be  reduced  to  $47.55.) 
Thus,  the  cost  per  sheet  on  large  runs  would  be  $7.59  f  1000  or 
.76  cents  per  sheet. 
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